| Oracle® OPatch User's Guide Release 12.1 for Windows and UNIX Part Number E39376-01 |
|
|
PDF · Mobi · ePub |
| Oracle® OPatch User's Guide Release 12.1 for Windows and UNIX Part Number E39376-01 |
|
|
PDF · Mobi · ePub |
This chapter describes common OPatchauto problem that may occur during usage.
This chapter covers the following:
In order for OPatchauto to fully automate the patching process, it accesses various tools/utilities to carry out different patching phases. The four primary tools/utilities are:
OPatch - Applies patches to GI/RAC/DB homes.
rootcrs.pl - Controls GI Home access by unlocking files so they are patchable, as well as stopping and starting instances to permit patching of GI files.
patchgen - Records the patch level.
datapatch - Applies SQL changes to database instances.
These tools/utilities are accessed during the patching process. Troubleshooting OPatchauto, therefore, involves diagnosing issues with the individual tools.
When using OPatchauto, problems may arise where it is not clear as to how to proceed with the resolution. The following use cases illustrate common patching scenarios where you may encounter such problems and general procedures you can use to resolve the problems.
The rootcrs.pl script performs the operations necessary to install the Oracle Clusterware stack on a node of a cluster. During an OPatchauto session, you may encounter errors stemming from the rootcrs.pl script.
Command issued by OPatchauto: $GRID_HOME/crs/rootcrs.pl -prepatch
If running rootcrs.pl fails , error codes and their associated messages will be generated, as shown in the following example
CRS-1159: The cluster cannot be set to rolling patch mode because Oracle Clusterware is not active on at least one remote node.
If the message is not clear, you can obtain additional help by running the OERR utility to obtain explicit cause and recommended action information.
Running OERR for a specific error code will generate both the cause and action for the specified error code. For example, running oerr for error code 1159 generates the following:
$GRID_HOME/bin/oerr crs 1159 Cause: The cluster could not be set to rolling patch mode because Oracle Clusterware was not active on any of the remote nodes. Action: Start Oracle Clusterware on at least one remote node and retry the 'crsctl start rollingpatch' command, or retry patching using the non-rolling option.
The following table list the common error codes that you may encounter during a patching session. For an exhaustive list, see the Oracle® Database Error Messages manual.
| Error Code | Console Message |
|---|---|
|
1153 |
There was an error setting Oracle Clusterware to rolling patch mode. |
|
1154 |
There was an error setting Oracle ASM to rolling patch mode. |
|
1156 |
Rejecting the rolling patch mode change because the cluster is in the middle of an upgrade. |
|
1157 |
Rejecting the rolling patch mode change because the cluster was forcibly upgraded. |
|
1158 |
There was an error setting the cluster to rolling patch mode. |
|
1159 |
The cluster cannot be set to rolling patch mode because Oracle Clusterware is not active on at least one remote node. |
|
1162 |
Rejecting rolling patch mode change because the patch level is not consistent across all nodes in the cluster. The patch level on nodes %s is not the same as the expected patch level [%u] found on nodes %s. |
|
1163 |
There was an error resetting Oracle Clusterware rolling patch mode. |
|
1164 |
There was an error resetting Oracle ASM rolling patch mode. |
|
1166 |
Rejecting rolling patch mode change because Oracle ASM is in [%s] state. |
|
1168 |
There was an error resetting the cluster rolling patch mode. |
|
1171 |
Rejecting rolling patch mode change because the patch level is not consistent across all nodes in the cluster. The patch level on nodes %s is not the same as the patch level [%u] found on nodes %s. |
|
1181 |
There was an error retrieving the Oracle Clusterware release patch level. |
|
1183 |
Oracle Clusterware release patch level is [%u] and an incomplete list of patches [%s] have been applied on the local node. |
|
1191 |
There was an error retrieving the Oracle Clusterware software patch level. |
Scenario 1: Non-rollable Patch is Applied in Rolling Mode
You have a two-node (node 1 and node 2) configuration and are attempting to apply a non-rollable patch in rolling mode.
Note:
By default, OPatchauto applies patches in rolling mode.Because you are applying the patch in rolling mode, you have not shut down all databases and stacks. When OPatchauto is run, it prints out the stack inventory and updates the binaries as expected.
When rootcrs.pl -postpatch (Performs the required steps after the Oracle patching tool (OPatch) is invoked) is run, it fails due to the library updates not being in sync. In this situation, OPatchauto (which runs OPatch) fails with a non-zero exit code. However, the patch is left in the GI Home. The stack cannot be brought up.
It is important to note that, in this situation, it is not necessary to roll back the patch as it has already been successfully applied to node 1. In general, make sure any attempt to bring up the stack is the very last step performed: Even if the stack fails to come up, the patch has been successfully applied to all nodes.
Because the patch is non-rollable, to resolve the stack issue:
Bring down the stack on all nodes.
Patch the remaining nodes.
Bring the stack back up on all nodes.
Scenario 2: OPatchauto Fails to Patch the GI Home
You have a system patch containing sub-patches (P1 and P2). When OPatchauto apply is run, it will first patch the RAC homes. In this scenario P1 is applied to RAC at time t1, P2 is then applied to RAC at time t2. OPatchauto attempts to apply sub-patch P2 at time t3 to the GI Home but fails.
OPatchauto fails with a non-zero exit code. The error message indicates failure occurred when applying sub-patch P2 on the GI Home. Note that the error message will provide you with a log file location. The RAC Home now contains P1 and P2, but the GI Home is missing P2.
You need to apply the missing patch to the GI Home. Because the system patch has already been successfully applied to the RAC Home, there is no need to roll back the patch.
From the log file, determine what caused patch application to fail for the GI Home.
Fix the issue that caused the GI Home patch application to fail.
When patch application fails for the GI Home, there are three possible causes:
patchgen -- In this situation, refer to the recommended action specified for patchgen use case. See "Patchgen".
You will have to manually patch the GI Home. Refer to the patch README for instructions.
opatch -call command failed. In this situation, an error occurred during OPatch execution. For example, OPatch could not copy a required file.
rootcrs.pl -prepatch (perform the required steps before OPatch is invoked) fails.
Regardless of the cause of failure, you must resolve the issue and then manually patch the GI Home.
Re-run opatchauto apply on the GI Home. OPatchauto knows where patch application has been successful and where it has failed.
When applying a system patch, OPatchauto fails as a result of error conditions encountered by patchgen.
OPatchauto fails with the STDOUT error message indicating a patching failure due to problems encountered by patchgen.
Determine whether the error message is a result of a patchgen error. From the message output, you can determine whether or not it is of patchgen origin by searching for the keyword "patchgen." The following example shows a sample error message generated by patchgen. The keyword "patchgen" and the associated error code is in bold.
Example 5-1 Patchgen Error Output
$export ORACLE_HOME=/scratch/GI12/product/12.1.0/crs $/scratch/GI12/product/12.1.0/crs/bin/patchgen commit -pi 13852018 loading the appropriate library for linux java.lang.UnsatisfiedLinkError: /scratch/GI12/product/12.1.0/crs/lib/libpatchgensh12.so (libasmclntsh12.so: cannot open shared object file: No such file or directory)
With the patchgen error code, run the oerr command to obtain the cause and recommended action(s) to resolve the specific problem encountered by patchgen. Implement the suggested action. See "Running OERR".
When patchgen errors out, it will ask whether or not you want to keep the patch or roll it back. By default, patchgen rolls back the patch. Whether or not the patch is rolled back determines your course of action in the next step.
If the patch was not rolled back, run patchgen again.
Despite the error, the patch itself still exists in the GI/RAC/DB home since it was not rolled back.
If the patch has been rolled back, you may find that the OPatchauto has applied the system patch to the RAC Home, but not all sub-patches to the GI Home. At this point, you need to apply only part of the system patch to the GI Home.
OPatch will tell you via lsinventory, which patches have not been applied. In order to apply specific sub-patches, you must resort to manual patching:
1. Shut down the stack.
2. Run opatch apply (not OPatchauto) on the GI Home.
Refer to the patch README for explicit instructions on applying a patch manually.
Table 5-2 Patchgen Error Codes
| Error Code | Reason | Debugging Information |
|---|---|---|
|
2 |
Internal Error |
Generic failure error code. |
|
3 |
Internal Error |
MS Windows: Resource file read error. |
|
4 |
Internal Error |
MS Windows: Resource file write failed. |
|
5 |
Internal Error |
Unix: Open for patch repository failed. |
|
6 |
Internal Error |
Unix: Normalization of full path libasmclntsh failed. |
|
7 |
Internal Error |
Unix: Write to patch repository failed. |
|
18 |
Internal Error |
PGA initialization failed. |
|
19 |
Internal Error |
Patch iterator init failed. |
|
40 |
Syntax Errors, appropriate message would be displayed. |
No argument to patchgen. Example: |
|
41 |
Syntax Errors, appropriate message would be displayed. |
No arguments to
Example: |
|
42 |
Syntax Errors, appropriate message would be displayed. |
-pi patchids are not numbers. Example: |
|
43 |
Syntax Errors, appropriate message would be displayed. |
-rb patchids are not numbers. Example: |
|
44 |
Syntax Errors, appropriate message would be displayed. |
Argument to
is something other than Example: |
|
45 |
Syntax Errors, appropriate message would be displayed. |
Patchgen invoked with invalid argument. Example: |
|
46 |
Loading libpatchgensh12.so failed. |
You attempt to run OPatchauto to patch four GI/RAC/DB homes. This patch contains both bits and SQL to update the database. When you run OPatchauto, it performs two actions:
Applies bits to the GI/RAC/DB home
Runs SQL (via the datapatch command)
Typically, you run OPatchauto on each GI/RAC/DB home. With each run, OPatchauto calls datapatch to run the patch SQL. datapatch will do nothing on the first three nodes (no-op). On the last (fourth) node, datapatch tries to execute the patch SQL.
If datapatch fails, you will see an error message. To find out if the error is from datapatch, view OPatchauto debug log. In the debug log there is a section for datapatch command execution that will have the diagnostic information.
You see an error message indicating OPatchauto has failed. This can be misleading as the origin of the message is unclear. The error message was actually generated by datapatch when it failed to apply the SQL to the last node.
In general, you can ignore the error message and then run datapatch manually on the last node. After running datapatch manually, run OPatchauto again.
Rollable VS. Non-Rollable Patches: Patches are designed to be applied in either rolling mode or non-rolling mode. For more information about rolling versus non-rolling, see "Patch Application Modes". Depending on whether the patch is rollable or non-rollable determines the course of action.
If a patch is rollable, the patch has no dependency on the SQL script. The database can be brought up without issue. Note that a rollable patch can be applied in either rolling or non-rolling mode.
If, however, the patch is non-rollable, then the patch must first be rolled back. Note that OPatchauto will prevent you from applying a non-rollable patch in rolling mode.
OPatchauto fails. Determine whether the failure is, in reality, caused by datapatch.
For rollable patches:
Ignore data patch errors on node 1 - node(n-1).
On the last node (node n), run data sync again. You can cut and paste this command from the log file.
If you still encounter datapatch errors on the last node, call Oracle Support or open a Service Request.
For non-rollable patches:
Bring down all databases and stacks manually for all nodes.
Run opatchauto apply on every node.
Bring up the stack and databases. Note that the databases must be up in order for datapatch to connect and apply the SQL.
Manually run datapatch on each node. Note that if you do not run datapatch, the SQL for the patch will not be applied and you will not benefit from the bug fix. In addition, you may encounter incorrect system behavior depending on the changes the SQL is intended to implement.
If datapatch continues to fail, you must roll back the patch. Call Oracle Support for assistance or open a Service Request.
Log files allow you to diagnose issues if problems occur during your patching session. OPatchauto generates log files for both the apply and analyze operations.
An OPatchauto session generates three types of log files.
Regular log file
Debug log file
Steps log file
The regular log file contains primary session information such as the GI/RAC/DB homes that were patched, the patches that were applied, and any specified patch options such as rolling or non-rolling.
The debug log file is a superset of the regular log file and is intended for use by Oracle support. In addition to including the content of a regular log file, the debug log file contains detailed session information. For example, for each command that is run the command output is also logged.
The steps log file contains only the steps that OPatchauto has generated during the patching session. This file allows you to view the steps OPatchauto has executed without having to wade through extraneous information. If problems occur during the patching session, you can see at which step patching process failed.
|
 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Legal Notices |
|
